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Book I: A Li nguistic Approach to Protocol Analysis 

1. Introduction 

1.1. SPADE: A Linguistic Theory of Design 

1.2. PATN: Analysis by Synthesis 

1.3. Theoretical Interpretations 

1.1. SPADE: A Lin guistic Theory of Desig n 

In recent research we have developed a theory of design called SPADE 
which provides a model of the planning and debugging processes. 1 We contend that, 
in addition to being a powerful theory of machine problem solving, SPADE is also 
a useful framework for describing human problem solving. To support this 
contention, we apply the SPADE theory to the task of analyzing problem solving 
protocols. 

^ By adopting this methodology we follow the precedent established in 

seminal protocol analysis studies conducted at Carnegie Mellon University 
[Newell 1966; Newell & Simon 1972; Waterman & Newell 1972, 1973; Bhaskar & 
Simon 1976]. Our work extends their approach along three dimensions. 

1. With the exception of the recent Bhaskar & Simon effort, the CMU 
studies have been restricted to very limited domains such as 

tZ P Zi thm % tiC ' Rath6r tha " limiting the tas * domain we limit 
«f\£?T ,° f J* es P° nses - Typically protocols are transcriptions 
of think-aloud verbalizations; we focus on the more restrict 
interactions arising from a problem solving session at a 
computer console. 2 The analysis task in this setting is to 
interpret user actions - editing, executing, tracing, etc -- 
in terms of the SPADE theory of planning and debugging 

Z ' I!i e / M ^ theory centers on ^e production systems model. Althouoh 
productions are Turing universal, they tend to result in a less 

of The^SPA^ 09 ^ organiz J! tion tha " ^e linguistic formalisms 
oL^ * AD 5 V 601 "*- The PATN program, the procedural 

nTtTrkfu °/ ?n\ e „f ADE the0ry ' uses an rented transition 
network [Woods 1970] to represent planning knowledge. 
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3. CMU analyses are based on the problem behavior graph. Pursuina 
an analogy to computational linguistics, we idefin" an 

i™?r; tat i° n ° f a Pr ° t0C01 t0 be a parse tree supplemented by 
tn? .nn^lt pr T atic "notation. The parse tree characterizes 
nrLm^, Ue f* StrUCtUre of the Protocol. Semantic and 
!o!« «f th n ° tatl ! n " variables and assertions attached to 
tit ration^i/fnt 6 tr V- "", formalize the Probl« description and 
trL[ m« , for P artlcular Planning choices. Annotated parse 
trees closely reflect the local structure of PATO's linguistic 
problem solving machinery, leading more directly to inferences 

ix^xt:^ 1 dif f erences than is evident *™z*& 

Ruven Brooks [1975] applied the CMU approach to the programming domain, 
developing a model of coding - the translation of high level plans into the 
statements of a particular programming language ~ and testing the model by 
analyzing protocols. His model is a set of production rules whose conditions 
match the patterns of plan elements and whose actions generate code statements. 
Protocols are analyzed manually, with the experimenter attempting to infer the 
^ Plan which is then expanded by the production system into code paralleling that 

of the protocol. The processes of understanding the problem, generating the 
Plan, and debugging are not formalized. SPADE goes beyond this in that it can be 
used to parse protocols and that the parse constitutes a formal hypothesis 
regarding not only the coding knowledge but also the planning and debugging 
strategies employed by the problem solver. 

The paper is divided into two books. Book I develops SPADE's linguistic 
paradigm for protocol analysis. A prototypical elementary programming protocol 
is parsed, and the implications of this information processing analysis for 
constructing cognitive models and designing computerized tutors are discussed. 

Book I does not address the question of how a protocol parse is derived. 

In earlier work, problem solving protocols were analyzed manually. 3 However, 

manual analysis is tedious and informal; hence Book II presents the design for 

^ PAZATN, an automatic protocol analyzer. PAZATN uses PATN 4 - the procedural 
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embodiment of the SPADE theory - as a generator for possible interpretations of 
the protocol, with bottom-up evidence biasing PATH toward plans which are likely 
to match the data (figure 1:1). 

PAZATN is a domain independent framework for constructing specialized 
protocol analyzers. To apply PAZATN to a particular task domain. event 
specialists (ESP's) are supplied which embody domain-specific knowledge. For 
concreteness. we employ examples from the Logo elementary graphics programming 
domain; ESP's for this domain are discussed. PAZATN's operation is hand- 
simulated on an elementary protocol from this domain. 

1.2. PA TN: Analysis by Sy nthasis 

A major insight of the generative grammarians (e.g., Chomsky [1965]) was 
that it is often helpful to characterize phenomena synthetically: one devises 
rules to generate the phenomena. Analysis can then be viewed as a recognition 
process for selecting derivations from the space of synthetic possibilities. We 
adopt this viewpoint in analyzing protocols, with PATN as our generative 
formalism. 

The SPADE theory, which PATN embodies, begins with a taxonomy of commonly 
observed planning techniques (figure 1:2). When a problem is confronted - 
according to the theory - one of three types of plans may be pursued. (1) The 
problem may be solved by identification: recognizing it as already having a 
solution. This planning category, seemingly trivial, is of course essential to 
avoid infinite regress. (2) The problem may be solved by decomposition: 
dividing it into smaller, easier subproblems. These are each solved separately, 
and then recombined, thereby disposing of the original problem. (3) Should the 
first two strategies fail, the problem may be solved by reformulation: 
^describing it in terms that seem more amenable to solution. The reformulated 
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problem must, in turn, be solved by identification, decomposition, or further 
reformulation. As the figure indicates, each of these categories of plans is 
further subdivided. 

PATH (figure 1:3) is a problem solving program based on this taxonomy. 
PATN was derived by first representing the taxonomy as a recursive transition 
network.* This produces a non-deterministic problem solver. Supplying precedence 
ordering for arcs from each node, predicates which test preconditions for 
transitions, and actions to be performed when arc transitions occur, produces an 
augmented transition network [Woods 1970] that is far more deterministic in 
solving problems (although backup is permitted). The predicates access registers 
which store semantic information about the problem; the actions modify these 
registers. 

PATN's solutions can exhibit rational bugs - errors arising from 
heuristically justifiable but incorrect planning decisions -- such as the trial 
execution of an incomplete plan, which omits necessary interface steps. Hence a 
complementary theory of debugging is developed using the same approach as in the 
planning theory. Figure 1:4 shows a taxonomy of debugging techniques. This 
taxonomy bifurcates into techniques for diagnosing the underlying cause of a bug 
and techniques for repairing the bug once isolated. Hodel diagnosis is typical 
of the diagnostic techniques. It consists of executing the program in order to 
construct a list of violated model predicates, which is then examined to check if 
any code was written to accomplish the violated predicates. As in the planning 
theory, the debugging taxonomy is transformed into an ATN (figure 1:5) by 
providing registers, arc predicates and so on. The debugging ATN is called DAPR 
(debugger of annotated programs), and is an integral part of the PATN system. 

Consider the operation of the planning ATN on an example from the Logo 
graphics environment, where students define, test, and debug procedures to draw 
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FIGURE 1*4 A TAXONOMY OF DEBUGGING TECHNIQUES 
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simple pictures. The WISHINGWELL (figure 1:6) is a typical beginner's project. 
The students' task description is a sketch of the desired picture. PATN, 
however, requires a formal task description: figure 1:7 illustrates this 
description, the model, for WISHINGWELL- Models are expressed in an assertional 
formalism developed by Goldstein [1974, 1975], which is similar to the first 
order predicate calculus. The model characterizes the range of pictures which 
match the sketch. 6 

PATN's solution to the WISHINGWELL task has three aspects: (1) an 
hierarchical plan derivation, summarizing the arc transitions which were 
followed; (2) a snapshot of the values of the ATN's registers attached to each 
node of the derivation, representing the semantic context at the time the node 
was created; and (3) a set of instantiated arc predicates at each node 
describing why the chosen arc transition was preferred to its competitors; these 
■ are called the pragmatic assertions of the node. 7 The semantic variables and 

pragmatic assertions relate the subgoal structure of the problem solving protocol 
to the model describing the task to be accomplished. 

Figure 1:8 shows PATN's hierarchically annotated solution. Naturally, 
this is not the only solution to the WISHINGWELL problem: to apply PATN to 
protocol analysis, we allow PAZATN to reject solutions which do not match the 
protocol data, forcing PATN to backup so that alternative solutions are 
generated. 

1.3. Theoretical Interpretations 

We define an interpretation of a protocol to be a PATN plan derivation: 
a parse tree whose fringe is the list of events (e.g., figure 1:8), augmented by 
annotation associated with each node of the parse. Since different plans 
sometimes lead to the same coding events, some protocols have more than a single 
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FIGURE 1:6 
WISHINGWELL PICTURE 
AN ELEMENTARY LOGO GRAPHICS PROJECT 
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Draw a WISHINGWELL with a square well and a triangular roof, 
connected by a pole which is a line. The roof should be above 
the pole, and the pole above the well. The well should be 
connected to the pole at the midpoint of the upper side of the 
well and the lower endpoint of the pole. The pole should be 
connected to the roof at the midpoint of the bottom side of the 
roof and the upper endpoint of the pole. The bottom side of the 
roof and the upper side of the well should be horizontal. 



(DEFINE-MODEL WISHINGWELL () 
(EXISTS (ROOF POLE WELL) 

(AND (TRIANGLE ROOF) 
(LINE POLE) 
(SQUARE WELL) 
(ABOVE ROOF POLE) 
(ABOVE POLE WELL) 
(EXISTS (P) 

(AND (CONNECTED WELL POLE (AT P)) 

(EQ P (MIDDLE (UPPER (SIDE WELL)))) 
(EQ P (BOTTOM (ENDPOINT POLE))))) 
(EXISTS (Q) 

(AND (CONNECTED POLE ROOF (AT Q)) 

(EQ Q (MIDDLE (BOTTOM (SIDE ROOF)))) 
(EQ Q (UPPER (ENDPOINT POLE))))) 
(HORIZONTAL (BOTTOM (SIDE ROOF))) 
(HORIZONTAL (UPPER (SIDE WELL)))))) 



Figure 1:7. Predicate Model for a Wishingwell 
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interpretation . The basic claim of this paper is that PATN can efficiently 

generate the psychologically plausible interpretations. 

Evidence for this claim rests on four sources of evidence. 

1. The heuristic adequacy of PATN as a problem solving program 
provides suggestive, though by no means decisive, evidence. At 
least for the restricted world of elementary Logo graphics, 
hand-simulations indicate that PATN is heuristically adequate. 

2. Introspection by human problem solvers is a weak but useful 
source of evidence. To some extent PATN was designed on the 
basis of introspection and hence has some support along this 
dimension. 

3. A strong source of evidence is the appropriateness of the replies 
of a question-answering module that performs retrievals and 
simple inferences over a database composed of these 
interpretations. The question-answering module is introduced in 
chapter two. The replies to the example questions given in that 
chapter seem appropriate to the authors. 

4. The strongest source of evidence is ability to predict 
performance in future situations on the basis of past behavior. 
Chapter three describes modifications to the ATN that provide 
predictive models of typical problem solving behaviors. 

We find this informal evidence sufficiently encouraging (as detailed in 
the remainder of Book I) to warrant the design (in Book II) of a precise 
framework for generating SPADE-style protocol interpretations. Future research 
will rigorously evaluate the psychological validity of these interpretations as 
follows. 

1. PATN will be implemented and tested on a broad range of examples. 
This will confirm its heuristic adequacy. 

2. An editor based on SPADE will be constructed as a structured 
programming environment, and transcripts of the problem solving 
behavior of programmers using this editor analyzed. Coupled 
with systematic interviews, this will provide evidence regarding 
the sufficiency of SPADE'S repertoire of planning and debugging 
concepts. 
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3. PAZATN with a question-answering interface will be implemented. 
The appropriateness of the replies generated by the question- 
answering module will be judged by skilled but unbiased 
informants, and by systematic subject interviews. 

4. A modeling component will be implemented that modifies the PATN 
ATN to be more in accord with a particular student's behavior. 
Tests will be conducted to determine whether the modified ATN is 
more successful in predicting performance on subsequent 
protocols. 

Before proceeding, a possible misconception involving the distinction 
between representational frameworks and psychological theories should be 
dispelled. Two hypotheses are defended by the research program outlined in this 
paper: (1) that ATN's are a useful representation for the models we are 
developing; and (2) that particular ATN's, the output of our modeling procedure, 
constitute theories of individuals -- stated in the language of ATN's — which 
make statements about the presence or absence of certain problem solving skills. 
Both hypotheses are of course subject to experimental verification. We do not 
argue that other Turing-universal formalisms (such as productions or Heidorn's 
[1975] augmented phrase structure grammars) cannot also represent these theories. 
Stronger claims regarding the validity of ATN's per se as psychological 
mechanisms require additional assumptions regarding processing costs and 
limitations which we are not currently prepared to defend. 
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2. An Example of Protocol Analysis as Parsing 

2.1. An Example Problem Solving Protocol 

2.2. Structural Description 

2.3. Semantics 

2.4. Pragmatics 

An analogy to computational linguistics has been fruitful both in 
defining the objectives of analysis and in designing the PAZATN system for 
automating the analysis process. The analogy suggests partitioning analyses into 
syntactic, semantic, and pragmatic components. These components correspond to 
the potential control paths, data flow, and branching conditions of a procedural 
problem solver. From a problem solving standpoint, these are modeled by the 
network of states and arcs, the registers, and the transition conditions of the 
augmented transition network. From a protocol analysis standpoint, syntax is 
f^ s represented as a parse tree; semantics and pragmatics are represented as 

annotation (variables and assertions) associated with each node of the parse 
tree. 

This chapter presents a SPADE interpretation for a typical WISHINGWELL 
protocol. Book II provides a hand-simulation of PAZATN generating this 
interpretation. 

2.1. An Example Problem Solving Protocol 

Since analysis consists of the selection of a PATN plan derivation, 
analyzing a protocol identical to PATN's default solution (figure 1:8) is 
trivial. Hence, a different protocol, involving a variety of plans including 
reformulation and repetition, serves as our example. 8 
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The student begins by writing an iterative procedure to draw the 
square WELL. 

E01 ?T0 WELL 

E02 >10 REPEAT 4 [20 30] 

E03 >20 FORWARD 100 

E04 >30 RIGHT 90 

E05 >END 

A superprocedure for the VISKINGWEa is defined by a sequential 
plan drawing first TREE, a previously defined procedure, and 
then WELL. 



E06 


?TO WW 


E07 


>10 TREE 


E08 


>20 WELL 


E09 


>END 



The WW program is executed, producing figure 1:9. 
E10 ?WW 

The program is edited to include an interface establishing the 
proper relation between TREE and WELL. 



Ell 


?EDIT WW 


E12 


>13 RIGHT 90 


E13 


>15 FORWARD 50 


E14 


>17 RIGHT 180 


E15 


>END 



2.2. Structural Description 

The result of analyzing this protocol is a data structure, the 
interpretation, consisting of syntactic, semantic, and pragmatic components. The 
syntactic component, diagrammed in figure 1:10, is the protocol's structure* 
description: a parse tree representing the sequence of PATN arc transitions 
required to generate it. 9 

Such structural descriptions capture one aspect of problem solving 
behavior. They provide a formal basis for answering questions regarding which 
plan types were used, a topic which could otherwise be discussed only 
intuitively. 10 Their most direct application is to answering "how questions." 



Protocol Analysis 



1.20 



Miller & Goldstein 



A\ 



\ \ 

\ 



TURTLE BEGINS HERE 



A / 



/ 



\ 



\ 



/"-\ 



•K 



> !i 



f 



TURTLE ENDS HERE 



FIGURE I;9 WW AT E10 — INTERFACE NEEDED 



Protocol Analysis 



1.21 



Miller & Goldstein 



o 
en 



W 

o 

En 
0- 



o 

CM 



w 



A 



o 
o 

rH 

Q 



o 
Eh 



O H 



o 

Cvj 

A 



O 

A 



W 

A 



rH CN CO 'vT LD 

o o o o o 

W H W pq pq 



O 
Eh 



Eh 



A 



o 

CN 

A 




a 
w 

A 



^D h 00 (^ 

o o o o 

W W W pq 



o 

rH 

w 



w 

•H 
CO 


c 

Cn 
fd 

•H 

i 

rH 

<u 





CO 
•H 
CO 

o 

G 



Eh 
H 
Q 
W 



o 

en 

O 
H 



O 

m 

Q 

o 



tf fa 



ro in 



A A 



o 

CO 

rH 

O 

H 



A 



a 
w 

A 

m 



rH CN On ^ 

rH rH rH pH rH 

W pq pq pq pq 



Eh 
H 



(1) 

> 

rH 


CO 

-P 

<D 

rH 

g 



u 



05 
0) 



w 

Q 
D 
En 
CO 

& 
O 

Ph 



O 

H 

H 

O 
CO 

w 

Q 

o 
u 

D 
P^ 
En 
CO 

Q 
W 
B^ 
< 
H 
> 

CQ 

PQ 
< 



CD 

> 

rH 


CO 



~3T 

W 
CO 

O 

o 



CD 
> 

rH 

o 

CO 



cq 

ID 
O 



c 
fd 

rH* 



D 

CD 
Q 



MH 
CD 



Protocol Analysis 1 # 22 



Miller & Goldstein 



/"*\ 



Ql . How does procedure WELL accomplish a square? 

Al. WELL uses a repetition plan. The generic subgoal is a side. 

Q2. How does procedure TREE accomplish a roof and pole? 

A2 * 2?«™«( „;? d P °J e ""^n 1 parts were "grouped into a tree by a 

r a l Plan> J rocedure TREE was alreai, y in the answer 
library, allowing an identification plan. 

Still, the parse tree is an incomplete description: it does not indicate 
the semantic relationships between subgoals or the pragmatic criteria governing 
the choice of one plan over another. 

2.3. Semantics 

Semantic annotation is defined to be the values of semantic variables 
associated with each node of the parse. These variables relate the plan to the 
formal problem model by recording the contents of the ATN's registers at the time 
the node was generated. The following are typical PATN registers. 

*' curf e ni\ t n^ hi r. r i Chi .? lly J ann0tated Pr09ram defined belo « ^e 
•virfr h "° d !' reflectin 9 its state after dominated editing 
events have been processed. M 

2. :CODE is the fringe of :PLAN. 

3 * riffTnoI £V d " cri P tion of the effect of executing the code 
defined below the current node. Since the code may contain 
references to undefined user procedures, :EFFECT may be 
unassigned at a given node. For the elementary graphics domain 
this variable is called :PICTURE, and describe the picture 
drawn by the program in Cartesian coordinates. Picture 

4 * iSSiuh th8 p S8t ° f predicates "hich :CODE is intended to 
?MODEL 3 C ° rreCt Pr ° 9ram :EFFECT is an instance of 

5. : ADVICE is a list of planning suggestions generated by PATN arc 
actions For example, the linearization arc (see PATM™ 
conjunction node in figure 1:3) creates advice regarding both 
the order in which subprocedures should be written, and the 
order in which they should be invoiced. 
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6 * DAT V M EAT f 1S v a list of warnin 9 s f °r potential bugs generated by 
PATN when heuristic guidelines are used in planning For 

iJSIivVnn n ° i nte . r , a . ctions are detected when solving a problem 
involving an unfamiliar domain predicate, it is possible that 

naltSrn^ 6 SCtUally 9iV6 rlS6S t0 interactions, but their 
•?AVEATS%r e H n0t / et b6 !, n learned by the **«*•■• Hence, 
irZ\±tinn f b r «- m ' ? cording this Potential bug, on the arc 
transition from conjunction to sequential plans This 

diagnosis?" 9Uid6S ° APR ' PA ™' S debu " in9 module ' in subsequent 

7 * sati^/il^hv li th i™ii* °u DOdel P redic *tes which are not 
"?F^CT ar « y JJ e K :EFFECT / ChieVed by :C0DE ' ni * re 9 ist er and 
Goldstllil f [1974]. a Perf0mance "Station module designed by 

Let us sample the values of the semantic variables at various nodes of 
the parsed WW protocol. :M0DEL for the top level SOLVE node was shown in 
figure 1:7. For the INT-TW SOLVE node, :M0DEL is: 

The tree must be above the well, and the bottom endpoint of the 
tree must connect to the midpoint of the upper side of the well. 

In our LISP-oriented model language notation this is represented as: 

(AND (ABOVE TREE WELL) 
(EXISTS (P) 

(AND (CONNECTED TREE WELL (AT P)) 

(EQ P (MIDDLE (UPPER (SIDE WELL)))) 
(EQ P (BOTTOM (ENDPOINT TREE)))))). 

This submodel reflects the reformulation of WISHINGWELL into a TREE and a WELL. 

Typically, semantic annotation is relevant to answering -what 

questions." The above value of .-MODEL for the INT-TW node provides an example. 

Q3. What is the purpose of lines 13, 15 and 17 of WW? 

A3 ' M°d"um ee Hh" aP ! in ," line COde int «f facing subprocedures TREE 
and WELL. The interface establishes connectivity at the 
appropriate point, and causes the tree to appear above the well 

: VIOLATIONS at the PLAN node for WW provides another example. 
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04 * event noT? " 9 "^ Pr ° CedUre W when " is first executed (at 

A4. The necessary relations between the model parts TREE and WELL 

SucS t°h a t be t e h n \ Stab \ ished: specifically, there is no Joint P 
such that the tree is connected to the well at P, P is the 

X! tree" 61 " ^ ° f the WeU ' a " d P is the lower •"dpoilt J? 

The VIOLATIONS variable which mediates this answer is non-empty at the PLAN node 
because the debugging which generates the missing interface has not yet occurred: 
the English answer simply paraphrases its LISP value: 

(NOT (EXISTS (P) 

(AND (CONNECTED WELL TREE (AT P)) 

(EQ P (MIDDLE (UPPER (SIDE WELL)))) 
(EQ P (BOTTOM (ENDPOINT TREE)))))). 

2.4. Pragmatics 

Pragmatic annotation is defined to be a record of the justifications for 
selecting a given arc transition over its competitors, and constitutes an 
hypothesis about the reasons for using a particular plan. REASONS are assertions 
attached to each node of the parse. The REASOU for using a particular plan in a 
particular situation is an instance of the arc predicate leading to the ATM state 
for that plan, where the current ualues of the registers are taken into 
account." For example, the reason that WELL was decomposed using a repetition 
Plan in the protocol is that :MODEL at that node was generic. 

(REASON (REPETITION E02) 

(GENERIC (:M0DEL E02))). 

Pragmatic annotation is germane to answering "why questions. - 

Q5. Why did the student execute WW at event E10 - did (s)he believe 
the program to be correct? ' oeneve 

A5. Probably the student expected bugs. A reasonable strategy is to 
initially plan only for the main steps, with the interfaces 
solved later by debugging. WW was executed at E10 in orSl? to 
discover what interfacing, if any, was needed. 
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This illustrates the analysis of a procedure containing the rational bug 

of constructing an incomplete plan." Debugging operations are analyzed by 

postulating the application of some DAPR technique. The reasons for debugging 

operations typically involve localizing or repairing the cause of some model 

violation. The purpose of running the program at E10 was to perform model 

diagnosis; this technique was chosen because the occurrence of two consecutive 

mainsteps (with no explicit interface) implies that the plan may be incomplete: 

(REASON (MODEL-DIAGNOSIS E10) 

(AND (OPTIONAL (INTERFACE TREE WELL)) 
(MISSING (INTERFACE TREE WELL) 
(:PLAN E06)))). 

In this case, model diagnosis demonstrates the existence of violated predicates 
for which no code exists: the plan is in fact incomplete. This is the reason 
for the subsequent editing: repair of the incomplete plan by resuming planning 
at the offending locale. 

The reason fo r the completio „ plan in the ediu episode (E12 
through E14) is to eliminate the violations by supplying the 
missing interface between TREE and VEU. "J»*in«ff "« 

(REASON (COMPLETE (E12 E13 E14)) 
(AND 
(MEMBER 
'(NOT 

(EXISTS (P) 

(AND (CONNECTED WELL POLE (AT P)) 

" ))) 
(VIOLATIONS E10)) 
(EQUAL 

'(EXISTS (P) 

(AND (CONNECTED WELL POLE (AT P)) 
" )) 
(: MODEL (INTERFACE TREE WELL))) 
(MISSING (INTERFACE TREE WELL) (:PLAN E10)))). 

The conjunction of predicates collectively called SEQ (on the arc from 
conjunction to sequential) plays a role in the following example. 
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Q6 ' Reverse? tHe inVOCation order < TREE WELL} used, rather than the 

A6. TREE ends at its bottom, a required connection point, resultina 
in simpler interfacing for that ordering. If the TREE btgal a? 

Jreferab n ief P ° int ' the reV6rS6 ° rder Would "ave been 

Here one of the SEQ predicates incorporates knowledge about the domain predicate 
CONNECTED, that interfacing can be simplified if two subprocedures are invoked in 
an order such that the endpoint of the first corresponds to a mutual connection 
point. An instance of this rule becomes a pragmatic assertion of the SEQ node in 
the parse. 

TME r ZZi?\/ SS r9 f er J in9 the {TREE WELL} se ^"dng is that 
JHEE ends at a required connection point of VELL. 

(REASON (SEQ (E07 E08)) 
(AND 
(EQ (POSITION : TURTLE (AFTER TREE)) 
f~S (BOTTOM (ENDPOINT TREE))) 

(EQ (POSITION : TURTLE (AFTER TREE)) 
(MIDDLE (UPPER (SIDE WELL)))))) 

A precise definition of a linguistic approach to protocol analysis has 
been provided and a concrete analysis of this kind supplied. We now turn our 
attention to the potential utility of the approach for constructing cognitive 
models of individuals. 
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i: — Toward a Co gnitive Model of the Individual 

3.1. Tailoring the ATN to the Individual 

3.2. Individual Differences and Overlay Modeling 

3.3. Issues and Examples, and the Computer as Coach 

3.1. Tailoring the ATN to the Individual 

Advocates of computer-aided instruction point out that computers can be 
used to tailor instruction to the needs of the individual. Yet little is known 
about what it means to construct cognitive models of individual students, or 
about how to use them in providing sensitive and effective automatic tutoring. 
The SPADE theory suggests an approach. 

SPADE confronts the problem of individual differences by considering the 
possible ways in which the student's ATN can differ from that of an expert. One 
error would be to have a variant of the optimal pragmatic arc constraints. More 
serious would be to have missing or extra arcs. Even more serious would be to 
have missing or extra states. Differences which can be formalized as alterations 
to the topology of the ATN are manifested in the production of a different set of 
parse trees: PATN might be capable of some derivations not available to the 
student, or vice versa. Differences in arc conditions or arc actions are 
manifested by the selection of other than the optimal plan for a particular 
problem situation, although the same repertoire of plans may be available. 

These types of modifications, properly combined, can account for many 
commonly observed weaknesses in student problem solving. To demonstrate this 
point, we present six examples of student weaknesses and the fashion in which our 
modeling scheme is able to capture them. The examples are derived from informal 
data collected in our prior Logo tutoring experiences. 
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X ' the JC BAM?'S!?m,.i. StUdent Wlth Pri ° r P r °9 ranmin g experience in 
the BASIC language never uses recursion. Problems for which 
iteration is awkward are solved only with difficulty; problems 
for which iteration is inadequate, such as drawing arbitrarily 
deep binary trees, are unsolvable. -ruitrariiy 

A deviant version of the PATN subgraph for repetition planning is 
illustrated in figure 1:11. The correct subgraph has an intermediate ROUND plan 
state; the deviant version, missing this state and its associated arcs, 
characterizes the BASIC syndrome. The student's repetition arc bypasses the 
ROUND state, short circuiting the ATN to pursue the iteration option with no 
possibility of recursion. In general, failure to employ a full repertoire of 
planning options can be modeled in this fashion: the short circuit is postulated 
to occur at the node immediately prior to the least common superset of the class 
of unused plans. 

2. Discontinuity: A student fails to build upon previous work 

Z e nW kin V d K Van i a9e ° f relevant listing' procedures Sach 
M £ re h t0 b ° drawn i« treated as an isolated problem, and 
recurring subproblems are repeatedly solved afresh. 

PATN can accomplish identifications using either primitives or previously 
solved problems. Discontinuity amounts to examining the primitive library only. 
This is modeled by the absence of the corresponding predicate on the arc from 
PLAN to IDENTIFY. A similar but pore subtle case would be a student that 
occasions* uses previous solutions, but not as often as PATN predicts. This 
indicates that the identification network is probably intact, but parts of the 
reformulation subgraph are missing. Such a student fails to notice the relevance 
of previous problems because they are described in slightly different terms. 
Introspection suggests that this is a common source of difficulty. 
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Dotted lines are missing arcs. 
Dotted circles are missing states 
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FIGURE I; 11 DEVIANT ATN SUBGRAPH FOR REPETITION PLANS 
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JlJf"?*" Avoidancei A student performs well in planning and 
?!i*L 9 P D r T amS u However - when a bug occurs, the student 
lllll n 'r +u tha r , s y stem atically localizing the underlying 

£«Jn. ♦ ,fr r °u' followed ^ re Pair, the student immediately 
begins to edit the program. The changes are haphazard and 
counterproductive, creating more bugs than they eliminate. 

Relative to the DAPR debugging ATM, diagnosis avoidance is a weakness 
wherein the student has an extra arc not present in the expert. Whereas DAPR 
cannot proceed to the REPAIR state without first passing through DIAGNOSIS, the 
student is modeled as having an undesirable extra arc bypassing this state 
(figure 1:12). This allows diagnosis to be (incorrectly) treated as optional. 

4. Syntactically Unstructured Code: A student never uses 
subprocedures, instead relying entirely on in-line code. This 
results in long, unreadable programs which are difficult to 
««?!! 9 ; e " the student forgets which subgoals have been 

i Vad „'. 0r , for 9 ets how previously solved code segments work. 
Few projects are successfully completed. 

PATN's use of subprocedures is governed by register setting actions 
associated with the sequential refinement loop. This is the culpable locale for 
a type of non-modular design we call syntactically unstructured code. Instead of 
first setting the .-PLAN register to a sequence of subprocedure calls, and then 
pushing for a solution to each in turn, the student apparently performs these 
actions in the reverse order: first pushing for a solution to each subprocedure, 
and then setting the :PLAN register to the concatenation of the popped results. 
Note that this deviant ordering of arc actions requires far more intermediate 
storage to keep track of recursive calls to the ATM: given a limited pushdown 
stack, it is not surprising that the student forgets things. 

5. Semantically Unstructured Code: A student mechanically begins 
every Logo procedure with the PENUP command. Usually this works 

whpn^hJ' in H , P reParatl ° n f ° r a position setu P' However even 
wnen the position setup is unnecessary, the PENUP is still used 

r\ resulting in either: (a) a rational form violation, in which 
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^ the PENUP is followed immediately by a PENDOWN; or (b) one or 

more missing model part violations, due to the invisibility of 
vectors intended to accomplish main steps. (Solomon [1976] uses 
the term cliche to describe this class of phenomena.) 

Treating some optional constituent as if it is required results in a 
second kind of non-modularity, semantically unstructured code. The particular 
clich& just described, mechanically including PENUP commands, is mediated by the 
linear decomposition arc. If this arc is modified to create position setup 
subgoals without testing whether :M0DEL actually requires such a setup, the 
effect is to include a PENUP at the start of each turtle program. 

6. Pragmatically Unstructured Code: A student who normally does 
break large programs into subprocedures nevertheless encounters 
numerous bugs, many of which are difficult to localize. The 
subprocedures lack modularity, each being dependent on knowledge 
of the inner workings of others. For example, interfaces are 
included as part of main steps, so that the initial state of a 
given procedure is determined by the final state of whichever 
^ procedure happens to precede it in the planned order of 

invocation. 

While failure of a particular arc action to consider the problem at hand 
results in semantically unstructured code, faulty arc predicates in deciding 
among alternative arcs leads to a third form of non-modularity, pragmatically 
unstructured code. The unnecessary construction of non-linear subprocedures is 
attributable to either improper default ordering or malfunctioning predicates on 
the arcs leaving the conjunction node. For example, the INTERACTIONS predicate 
may not be imposing sufficiently strong conditions on accepting the model: this 
leads to the addition of constraints on the subprocedures when no real non- 
linearity is present. 

Thus perturbations on PATN provide a deep theory of student weaknesses, 
explaining unsuccessful behavior in terms of the syntactic, semantic, and 
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pragmatic structure of the ATN. 

3.2. Individual Differences and Overlay Modeling 

We envision inducing a model of individual idiosyncrasies as 
perturbations of PATM by applying Goldstein's [1976] overlay modeling technique. 
This approach describes individuals with respect to an expert problem solving 
program by associating probabilities with each decision point in the expert, 
representing our state of knowledge about a given individual's preferences. The 
probabilities are a summary of the available evidence rather than an integral 
part of the model: at any given time, a process model is obtained from the 
overlay probabilities by including those possibilities that are above threshold 
and excluding those that are below. 13 Goldstein and Carr [1977] use this 
technique to infer process models of behavior in a logic and probability game 
called WUMPUS. 

This raises the question of whether all of the perturbations mentioned 
above, including the alterations in ATN topology, can in fact be represented by 
such an overlay, i.e., by a numerical plausibility table: it turns out that they 
can. A missing arc can be handled by assigning it an a priori transition 
plausibility of 0. A missing intermediate state can likewise be represented by 
the plausibility of the arcs leading to the unused states being 0, but the 
plausibility of the arc to the "short circuited" state being 1. Similarly, 
default orderings can be reversed by reversing their relative plausibilities. 

This table driven organization allows distinguishing between personal and 
archetypal ATN's. Archetypal ATN's are analogous to Winston's [1970] concept 
models, and in fact our scheme for inducing personalized ATN's bears some 
resemblance to Winston's learning system, except that our networks happen to have 
procedural rather than structural meanings. Personalized ATN's are created from 
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the archetype by thresholding over a particular plausibility table. This 
simplifies the tuning and debugging of the system and eliminates the danger that 
states or arcs added to model non-expert behaviors might degrade the performance 
of the expert. The expert ATN is obtained by coupling the archetype with an 
expert plausibility table. The entries for undesirable options are zeroes. 

A first approximation to the plausibility table for an individual can be 
derived from the relative frequencies of arc transitions in previously analzyed 
protocols. However, this ignores the semantic and pragmatic context. It could 
be that infrequently used transitions were inappropriate for the tasks performed. 
Consequently, this is refined by comparing the tallies to a record of the 
expert's performance over the same set of tasks. (This technique, differential 
modeling, is suggested by Burton & Brown [1976].) Naturally there will be 
differences in individual protocols because of arbitrary choices, but in the long 
term consistent properties of the student's behavior should emerge. 

Just recording arc transitions is still too crude. One should account 
for differences in terms of the smallest chunks of malfunctioning knowledge which 
can be isolated. As a second order cognitive model, the units of analysis are 
taken to be the individual arc predicates and actions. The statistical evidence 
can be used to differentiate which arc operations are malfunctioning or missing. 

3.3. Issues and Examples, and the Computer as Coach 

Two crucial ingredients are lacking in current uses of computers in 
education: a cognitive theory describing the problem solving and learning 
processes, and a pedagogical theory prescribing techniques to facilitate and 
enhance these processes. As a result, many instructional applications of 
computers are ad hoc, if not detrimental. 

There are exceptions to these criticisms. The Logo project [Papert 1971] 
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offers educational applications of computer technology suggested by a 
computational approach to problem solving and learning. However, the 
justification for many Logo insights remains informal and intuitive. The current 
work is an effort to increase the theoretical precision and experimental rigor of 
Logo research. Other exceptions include the work of John Seely Brown's group 
[Brown et al. 1974,1975; Burton & Brown 1976] on intelligent instructional 
systems for electronics (SOPHIE) and elementary mathematics (WEST), and that of 
Stansfield, Carr and Goldstein [Stansfield et al. 1976; Goldstein & Carr 1977] on 
an advisor for WUMPUS. The WEST tutor suggests a paradigm, also used in WUMPUS, 
in which issues (abstracted differences between expert and novice behavior) are 
illustrated by concrete examples of their application to active learning 
situations. 

Given the cognitive modeling tools developed in this chapter, an issues- 
/^ and-examples Logo tutor can be contemplated. When PATN's expectations are 

violated because of a difference between the expert and student versions of the 
ATN, then that issue can be raised with the student. This would extend the 
issues-and-examples paradigm of WEST and the computer-as-coac/> paradigm of 
WUMPUS, not only by addressing a more difficult task domain, but also by 
elaborating the notion of issues, from abstractions of empirically selected 
features, to specific programmatic weaknesses. 

The theory would also constrain the order in which issues should be 
presented to the student. The topology of the ATN should be nearly right before 
pragmatic arc constraints are discussed. Likewise, the general form of the 
pragmatics should be correct before domain-specific arc critics are taught. 
Although many subtleties arise which are not touched on here, the approach takes 
a step toward theoretical foundations for computer tutors which provide 
/"■"s sensitive, flexible, individual instruction in problem solving skills. 
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4. Notes to Book I 

1. SPADE is an acronym for Structured Planning and Debugging. See 
[Goldstein. & Miller 1976a, b; Miller & Goldstein 1976a, b,c]. 

2. More accurately, the session transcript is a partial protocol. 
Considerable leverage is obtained by assuming that the dialogue occurs within the 
confines of a small, well-defined response menu: natural language processing 
need not be attempted. We recognize that thorough protocol analysis includes 
parallel examination of the subject's utterances during the session, eye movement 
data, retrospective accounts, and so on. Although our sole objective here is 
analysis of the session transcript, we intend to corroborate our analyses using 
these other sorts of evidence. 

3. Miller & Goldstein [1976b] used a context free problem solving grammar 
to extract the constituent structure of a student's Logo protocol. That paper 
did not develop the more thorough view of analysis we describe in Book I of the 
current report. 

4. PATN is designed in [Goldstein & Miller 1976b]. It has not yet been 
implemented. The use of present tense throughout this document in describing both 
PATN and PAZATN is for readability only. 

5. For efficiency, some states with similar topology are merged, and a 
few additional arcs are added to provide for such features as iterative control, 
when recursively invoking the complete system is unnecessary. 

6. The figure adopts a parenthesized notation (which is formally 
equivalent to that used in our earlier papers) to emphasize that predicate models 
are just LISP S-expressions which can be evaluated. 

At first these predicate models will be supplied by the experimenter. 
Eventually we plan to construct a module to induce the model from a hand-drawn 
tablet sketch. A significant undertaking itself, this would enhance the 
practicality of automatic protocol analysis in the graphics domain. 

7. Generation of pragmatic assertions representing instances of arc 
predicates is an elaboration of the basic PATN design, not presented in 
[Goldstein & Miller 1976b]. These assertions, being directly computable by 
examining the ATN's arcs and the semantic variables, are synthetically redundant, 
but become important when analytic complexities such as irrational bugs and 
personalized ATN's are considered. 

8. The example is a simplified hypothetical protocol not involving 
careless errors such as mistypings. In other respects, however, it is typical of 
student protocols for tasks similar to WISHINGWELL. 
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9. The root of the parse tree is shown at the left; the leaves are to 
the right. Some details are not shown: ellipses are indicated by three periods. 
For clarity, some semantic information is included parenthetically: SOLVE(WELL). 

Logo punctuation events are of minor importance in the underlying plan. 
Although used as clues during parsing, they are not included in the structural 
description. In the figure they are shown enclosed in exclamation marks: 
!E05 >END! . 

Since the order in which subgoals are solved need not mirror their 
execution order in the resulting plan, events need not occur in the parse in 
temporal order. In the figure the events are shown in temporal order, but lines 
are crossed. 

10. To illustrate the insights gained from the analysis, we use a 
scenario for a question-answering module which performs retrievals and simple 
inferences over a database consisting of the analyzed protocol. We are confident 
that the data structures generated by our style of analysis are sufficient to 
support this type of interaction. However, we have not yet designed the 
question-answering module per se; instead, we have concentrated on isolating the 
relevant knowledge base. For readability, the questions and answers are stated 
here in unrestricted English; for ease of implementation, the actual system will 
be restricted to a formal query language. 

11. It might seem that this definition of pragmatic annotation is 
inadequate for protocol analysis, since a student may select the right plan but 
for the wrong reason. The SPADE approach handles this circumstance by a separate 
mechanism, personalized ATM's, to be discussed shortly. For ease of 

/-\ presentation, the example uses the expert ATN as the basis for its REASON 

assertions. 

12. Although PATN's default solution to the wishingwell task did not 
involve debugging, PATN is capable of rational bugs such as this particular 
incomplete plan. When solving novel tasks, it is sometimes more efficient to 
plan only for the main steps, with the interfaces being solved by subsequent 
debugging. During planning, PATN notes those points where the plan is 
incomplete; when a bug is encountered, this advice guides PATN's debuqaina 
module, DAPR. * 

Not all rational bugs are incomplete plans, and not all bugs are 
rational. Overlooking an interaction between subgoals is another type of 
rational bug. Mistypings and mispellings are typical irrational bugs; our 
approach to their analysis should be mentioned. The reason for such an event is 
assumed to be the same as the reason for the correct version of the event, but 
flagged by an additional assertion stating the nature of the mistake. 

13. Of course, one can also use probabilities to model actual non- 
determinism in the subject's behavior, but we do not consider that possibility 
here. 
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Book II; Automating the Protocol Parsing Process 
5. Introduction to Book II 

5.1. The CMU Series of Analyzers 

5.2. Overview of PAZATN 

Book I developed the SPADE notion of protocol analysis as parsing, but 
did not indicate how parses are to be derived. Automating the analysis process 
is desirable, because manual analysis is informal, tedious, error prone, and not 
amenable to incorporation into computerized tutors. Hence, this second book 
presents the design for PAZATN, an automatic protocol analyzer based on the SPADE 
theory. As background for assessing the design of PAZATN, we first summarize the 
features and limitations of a series of automatic protocol analyzers developed at 
/■■s Carnegie-Mellon University. 

5.1. The CMU Series of Analyzers 

Much ground-breaking research in automatic protocol analysis has been 
performed at Carnegie-Mellon University. PAS-I [Waterman & Newell 1972], the 
first of three CMU systems, analyzes think-aloud protocols for cryptarithmetic. 
PAS- 1 1 [Waterman & Newell 1973] is an interactive version which makes fewer task- 
specific assumptions. SAPA [Bhaskar & Simon 1976] addresses the additional 
complexities of semantically rich task domains. 

By focusing on the cryptarithmetic task, PAS-I obtains sufficient 
leverage to completely automate the analysis process. The input to PAS-I is a 
transcription of a tape recorded think-aloud protocol and its output is a problem 
behavior graph. PAS-I operates in four stages, the first two of which occur 
sequentially in time: linguistic analysis, semantic analysis, processing of 
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operator groups, and problem behavior graph generation, pas- I does not attempt 
generality; for example, the linguistic analyzer employs a key word grammar 
oriented to cryptarithmetic. Similarly, rrocess-CoZumn is a typical operator. 

PAS-II reduces dependency on a single domain by requesting guidance- from 
a human encoder. Task-specific knowledge is factored into a separate set of 
rules; the domain independent part of the system amounts to a command language 
or subroutine library to assist a human protocol-analyst. Moving from automatic 
to interactive analysis may seem counter to progress. However, this 
methodological contribution allows flexibility to incorporate the experimenter's 
insight, while still imposing discipline on the encoding process. We intend to 
construct an interactive analyzer as an intermediate milestone in implementing 
PAZATN. 

SAPA, in cooperation with a human encoder, analyzes protocols in chemical 
engineering thermodynamics. By considering a domain rich in background 
knowledge, rather than puzzle problems such as cryptarithmetic, SAPA addresses a 
complex new facet of problem solving. However, SAPA is highly domain specific. 
For example, SAPA begins the analysis by asking for the form of the energy 
equation used by the subject. Thermodynamics problem solving is viewed as a 
variant of means-ends analysis in which the energy equation plays a predominate 
role. 

When implemented, PAZATN will extend the automatic protocol analysis 
techniques developed at CHU by complementing their features and limitations. On 
one hand, a PAZATN shortcoming - its restriction to a small nenu of responses ~ 
is addressed by the considerable effort CMU researchers have invested in natural 
language front-ends for protocol analysis. On the other hand, CMU has devoted 
less attention to the investigation of planning concepts, a limitation addressed 
by the SPADE theory. For example, the CMU theory does not provide a deep account 
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of the origins of planning errors in the PATN sense. Likewise, a practical 
limitation of the CHU analyzers has been task specificity. In designing PAZATN 
we have tried to minimize task specificity through modular design; this is made 
Possible, in part, by the highly structured underlying SPADE theory. However, 
testing the generality of PAZATN by applying it to several domains remains a 
research goal.' Finally, the elementary programming world to which PAZATN is 
applied in this paper resembles thermodynamics in that background knowledge of 
the domain plays a significant role in solving problems. 

5.2. Overview of PAZATM 

PAZATN is a scheme for matching a protocol to a PATN plan derivation; it 
can only understand protocols which PATN can generate.* Therefore the analysis 
could be performed, in principle, by trying all possible PATN solutions, 
selecting the first which matches the data. Since exhaustive enumeration is 
impractical, a primary consideration is efficient search in PATN's plan space. 
Bottom-up protocol evidence is used for this purpose (figure 1:1). 

PAZATN consists of PATN supplemented by several additional modules and 
data structures (figure 11:1). This design incorporates three key ideas: 

1. the use of the chart data structure [Kay 1973; KaDlan 1Q711 in 

^^"^.r 0165 ' b0th Evolving 'the need to economically 
store alternative combinations of substructures; economlcal iy 

2. the use of a library of domain-specific specialists fnr 
processing events in various syntactic categories; 

3. the use of best first coroutine search driven bv a ™ n *r*t a 
scheduler - with modules communicating by means of the charts 
- to ensure early application of strong sources of constraint 

1. Two charts, One of PAZATN's charts, the planchart, keeps track of 
subgoals proposed by PATN. PAZATN's second chart, the datccAart, records the 
alternative ways of associating protocol events with planchart leaves. 
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2. iioran, o/ event specialists, The syntactic classification of 
possible events for a given domain results in a highly modular design. (ADDCODE. 
RUNCODE and END are typical Logo event types.) For each event type PAZATN is 
supplied with an ESP, i.e., a specialist for associating events of that type with 
Planchart leaves. Adapting PAZATN to other problem domains is possible by 
replacing this library. 

3. Best first coroutine search: PAZATN receives the model and protocol 
as input. The model is a formal statement of the problem as shown in figure 1:7; 
the protocol is a list of events as shown at the top of page 1.19. PAZATN 's 
output is a SPADE interpretation of the protocol as described in Book Is a parse 
tree augmented by semantic and pragmatic annotation. At any given time during 
analysis, several partial interpretations will be active. The outer loop is a 
scheduler which allows each active partial interpretation to examine one event 

r*> Per cycle. For a given interpretation, events are processed in a single left-to- 

right pass. At the end of a cycle the active set is re-chosen. This repeats 
until at least one interpretation has processed the final event. 

Analysis of a protocol proceeds as follows. First PAZATN requests PATN 
to generate its most plausible plan on the basis of the model alone. This plan 
is inserted into the planchart. Next, protocol events are examined one by one, 
matching them with subgoals in the PATN plan. Each match is recorded in the 
datachart. 

If an event is encountered for which no plausible match can be found, 
PATN is asked to generate its next most plausible plan, now potentially 
considering the nature of the mismatch as well as the model. The planchart is 
extended by inserting PATN's next plan. Those subgoals which are common to both 
plans share the same structure in the chart. 
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If an event is encountered for which more than one plausible match can be 
found, the datachart records each such pairing in similar fashion. Each of these 
is then allowed to continue examining protocol events according to the best first 
scheduling algorithm. 
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6. Simulating Automatic Parsing of the Example Protocol 

6.1. Preliminary Generation of Expectations 

6.2. Modifications Based on Bottom-up Evidence 

6.3. Some Informal Observations 

This chapter is a hand-simulation of PAZATN on the WISHINGWELL protocol 
introduced in Book I. Various components of PAZATN are introduced as they are 
needed. Subsequent chapters provide details regarding these components. 

6.1. Preliminary Generation of Expectations 

The protocol parsing process is initiated by executing (PAZATH 
WSHINGUELL Ml), where WISHINGWELL is the model and WW the protocol. The initial 
answer library is assumed to contain procedures for TRIANGLE and TREE. 

Before PAZATN examines the protocol, PATN examines the model. Since 
WISHINGWELL is not in the answer library, PATN determines that an identification 
plan is not viable. Both decomposition and reformulation are possible, since 
they are applicable to any model. 

PATN can determine, using lookahead, that reformulation results in an 
identification involving TREE; for this particular protocol, this quickly leads 
to a successful parse. However, reformulations rapidly expand the search space, 
so PAZATN adopts a conservative approach to reformulation: decompositions which 
lead to a straightforward solution are preferred unless protocol evidence 
indicating reformulation is discovered. Consequently decomposition is predicted, 
with three main steps: ROOF, POLE, and WELL. But since the decision is 
uncertain, a demon procedure 3 is created to handle the possibility that 
decomposition fails to parse the protocol. 

The model is examined for interactions. None are detected, so a linear 
decomposition into subgoals is expected. However, since required connection 
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^ points occur at the midpoints of sides of WELL and ROOF, a non-linear subgoal 

decomposition might be used for efficiency, to avoid retracing. As a result, two 
demon procedures are created to check for WELL or ROOF sides being accomplished 
in two steps. Such a plan is less likely for ROOF which can be identified with 
the existing TRIANGLE. 

The transitive ABOVE predicates suggest a sequential plan utilizing 
cither the order, {ROOF POLE WELL}, or the order, {WELL POLE ROOF}. There is no 
basis for selection. Hence, PATN follows a principle of least commitment, 
predicting the disjunction of the two invocation orders. 

This application of the principle of least commitment is accomplished 
using a cftart* of alternative plan derivations called the planckart. The 
Planchart is similar to an AMD/OR goal tree but involves a variety of node types 
and shares substructures economically. Figure 11:2 illustrates how the two 
^ equally likely sequences are represented in the planchart. As PATN generates 

predictions, the required bookkeeping is performed by expanding this planchart. 

Since the main steps for the two sequences are identical, they provide no 
evidence regarding ordering. The interfaces provide the critical evidence, so 
PATN solves the interfaces for one order, {WELL POLE ROOF}. Because the choice 
is arbitrary, another demon is created to expand the {ROOF POLE WELL} order in 
case the interfaces fail to match. Except that TRIANGLE is already in the answer 
library, PATN has predicted the protocol of figure 1:8. 

Besides predicting PATN's default solution, three arbitrary choices have 
been flagged as likely failure points. If the specific discrepancy pattern 
monitored by one of the three corresponding denons is detected, that choice will 
be reconsidered. If non-specific mismatches are encountered, backup to other 
decisions will occur in the usual way. Note that most choices are not arbitrary 
^ and have *ot been flagged. (This helps to avoid the usual inefficiency 
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associated with pure backtracking control structure: -failing in all possible 
ways.") 

At this point, control passes to the bottom-up analytic routines. 

6.2. Mo difications Based on Bottom-up Evidence 

PAZATN now attempts to interpret the first protocol event in a manner 
consistent with PATN's default solution. 

E01 ?T0 WELL 

E01 is classified as a TO event - Logo punctuation beginning a procedure 
definition. The event specialist for TO events is called upon to assign the 
event to some expectation. 

PAZATN does not use mnemonic clues, and no significance is attached to 
the student's particular choice of the name WELL. 5 The TO specialist examines the 
planchart (figure 11:2) for candidate subprocedures. There are expectations for 
the top level (WW), WELL, POLE and the two interfaces. The default solution 
order is top-down, so E01 is assumed to start WW. However, solution order is so 
variable that other interpretations are plausible. Consequently the 
interpretation splits into separate analyses for each. 

Whereas the planchart is used to keep track of alternative expectations, 
a second chart, the datachart, is used to keep track of alternative associations 
between protocol events and expectations. PATN expands the planchart; PAZATN 's 
event interpreter expands the datachart. At any given time, some of the partial 
interpretations in the datachart are considered to be active; the rest are hung. 
For expository purposes, we will assume that only one partial 
interpretation is active at a time. Rather than pursuing several alternatives in 
parallel, we will merely record them in case the need to back up arises. Hence 
after the split is performed, E01 is assigned to be the TO for the top level 
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procedure, WW. The parent node of this interpretation, a generator for 
alternative interpretations of E01, is hung. 

With E01 assumed to start WW, E02 is now processed. 
E02 >10 REPEAT 4 [20 30] 

This does not match the expectation for a definition of the top level WW 
procedure. Therefore, backup occurs to the most recent split (at E01). The only 
alternative that can account for E02, that E01 is the start of WELL, is activated 
(figure 11:3). 

The protocol matches this new interpretation through E05. 

E03 >20 FORWARD 100 

E04 >30 RIGHT 90 

E05 >END 

Ambiguity arises at E06. 
E06 >TO WW 

/•^ Since ROOF can be identified with TRIANGLE and WELL has already been found, this 

must be WW, POLE or an interface. The POLE and interfaces are apt to be solved 
by in-line code; furthermore, top-down order is the default preference. Hence, 
although E06 causes a split, WW is clearly chosen as the active interpretation. 
Next, E07 is examined. 

E07 >10 TREE 

Rather than matching WW's expectations for a setup or a call to WELL, E07 
matches the discrepancy pattern for two active demons. One demon represents the 
possibility that the {ROOF POLE WELL} order was used; this would require TREE to 
be the setup for ROOF. The other demon represents a potential reformulation 
involving TREE. This second demon is highly specific for this evidence and is 
therefore triggered. Control returns to PATN with a request for a reformulated 
model in which TREE is a subgoal. 
^ PATN regroups ROOF and POLE into TREE, and then expands for a solution to 
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the revised model. When the sequential refinement loop is reached, the 
{TREE WELL} invocation order is chosen immediately on the basis of known protocol 
data. This need not have been PATN's choice from a problem solving point of 
view: this decision is forced by the bottom-up evidence. Figure 11:4 shows the 
modified planchart. 

After PATN has processed the reformulation request, E07 can be 
accommodated, as shown in the datachart of figure 11:5. EOS is now examined. An 
interface is expected. 

E08 >20 WELL 

WELL is known to be a previously solved mainstep, violating that expectation. 
This is the standard pattern for an incomplete plan: an interface is expected 
but instead the next mainstep is found. A demon for incomplete plans is always 
active and is triggered by this situation. It passes control to DAPR which 
generates debugging expectations. 

Each remaining event matches a DAPR expectation. 

E09 >END 

E10 >?WW 

Ell >?EDIT WW 

E12 >13 RIGHT 90 

E13 >15 FORWARD 100 

E14 >17 RIGHT 180 

E15 >END 

Hence the parse succeeds. Figure 11:6 shows the final planchart and datachart, 
with marked nodes indicating the parse tree which is returned. 

6.3. Some Informal Observations 

We have hand-simulated PAZATN on about a half-dozen hypothetical 
protocols. This informal exercising of the design has led us to a number of 
tentative observations regarding PAZATWs capabilities. One question which 
arises is PAZATN's flexibility to handle alternative solutions. We are confident 
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that the following types of variation can he handled: 

1. subprocedures versus in-line code; 

2. incomplete plans where interfaces are solved by debugging; 

3 ' (SSTSn^ASr int6raCti0nS are overlo °^ ^ the student 

4. permutations of invocation or solution order; 

5 * cVnve?s a i r o d n); ref0rBUlati0nS (re 9 rou P ln 9> generic-explicit 

6 ' ef7i e c C iencJ) r ; y nonlinear ^compositions (accidental or for 

7. non-standard default parameters (FORWARD 75 as the basic unit); 

8. simple forms of equivalence (BACK 100 versus FORWARD -100); 

9. common errors such as mistyping, or omission of a line number. 

On the other hand, the following types of variation pose problems for PAZATN: 

1. interleaving of lines from different procedures if errors also 
occur in that a procedure is accidentally edited; 

2. unrecognizable reformulations due to gaps in PATN's knowledge; 

3 ' S.VJuon'.V 7 ° bSCUre C0d6 ' ° r C0d8 lnVOlvin « nan * "••«•« 

4. equivalence transformations resting on subtle domain theorems; 

5. fully general recursion including heterarchical procedure calls. 

Another observation concerns PAZATN's efficiency. For the simple 
protocols we have considered, after only a few false starts, PAZATN latches onto 
a correct set of expectations regarding the student's overall plan. After that 
Point (which we would place at E08 for this protocol) interpretation of the 
remaining events proceeds without incident. 
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L Organization of the PAZATM Protocol Parser 

7.1. . The Planchart 

7.2. Representing Interpretations 

7.3. The Datachart 

7.4. Incremental Planchart Expansion 

7.5. Markers and Marker Propagation 

7.6. Preprocessing 

7.7. The Event Classifier 

7.8. The Event Interpreter 

7.9. The Event Specialists 

7.10. The Scheduler 

In generating potential protocol interpretations, PATN is guided not only 
by synthetic evidence derived from examining the model, but also by analytic 
evidence derived from previously examined protocol events. If previous events 
have established that the student is pursuing a particular subgoal. then PATN 
will propose candidate solutions for that subgoal, even if it is not one which 
arises in PATN's preferred plan. Likewise if previous events have established 
that the student is pursuing a particular invocation order, then PATN will use 
that order in creating interfaces, even if another sequence leads to simpler 
interfaces. This sensitivity to the student's plan is accomplished by adding 
additional predicates to PATN's arcs which access assertions in the current 
partial interpretation. 

This chapter presents the major PAZATN modules needed to use PATN in this 
analytic role. Chapter eight refines the discussion presented here. 

7.1. The Planchart 

PATN is an intensional representation of the plan space; there are a 
number of reasons for needing an extension^ representation of the ATN process. 
Consequently a complete trace of PATN's operation, the planchart, is maintained. 
One reason for creating this data structure is to avoid repetitive calculations, 
but additional uses for the planchart will appear in the course of the 
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discussion. (Figure 11:4 shows an example planchart from the analysis of WW.) 

The planchart includes not only plans, but nodes of other types such as 
debugging episodes. As its name suggests, the planchart is a cftcrrt [Kay 1973; 
Kaplan 1973], a network which compactly represents alternative combinations of 
subexpressions. This economically represents PATN's partial solutions and their 
hierarchical annotation. Rather than generating the entire solution space at 
once -- which would be impractical even if it happened to be finite -- PATN 
expands this planchart incrementally as additional possibilities are needed by 
the analyzer. 

Looking upward from a given leaf, the planchart resembles an AND/OR goal 
tree. However, there are a greater variety of node types, rather than just AND 
and OR. This allows the planchart to represent such concepts as whether 
conjunctive subgoals need to be accomplished in a specified order, or whether any 
order will do, allowing a greater variety of potential interpretations to be 
expressed parsimoniously. 

The analysis process is closely tied to modifications of this data 
structure. In particular, the structural description assigned to a protocol 
corresponds to a pathway through the planchart starting from the root — the top 
level SOLVE node -- to the individual protocol events corresponding to a subset 
of the leaves. The semantic variables and pragmatic assertions are generated by 
PATN along with the parse, and are attached to the corresponding planchart 
nodes. Consequently, the structure building actions of the protocol parser are 
performed entirely by PATN. 
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7.2. Rep resenting Interpretations 

An interpretation of an event is represented as an assignment of that 
event to a leaf of the planchart (figure 11:7). Similarly, an interpretation of 
the protocol is a complete association list of such event assignments. A partial 
interpretation is an association list containing assignments for a subset of the 
events in the complete protocol. 

Because of the chart representation of plans, individual events can be 
assigned to a single leaf but remain ambiguous as to which plan they belong to. 
The assignment captures exactly what can be concluded from the event: no more 
and no less. All possible interpretations consistent with the data are carried 
along. 

In order to be assigned to a given leaf of the planchart, it is not 
necessary for the protocol event to match identically. Data events are converted 
to canonical form before assignment, so that equivalent forms (e.g., LEFT 90 and 
RIGHT 270) are not distinguished. Non-equivalent assignments are also possible, 
representing the analyzer's judgment that the protocol event was intended to 
match the planchart leaf but contains either errors, such as mispellings or 
mistypings, or different default parameters where a range of values is 
acceptable. 

7.3. The Datachart 

A partial interpretation splits when it proposes more than a single 
planchart assignment for an event. Some method for keeping track of the 
analyzer's alternative partial interpretations is needed. It should take 
advantage of the fact that, following a split, the event interpretations prior to 
that split remain the same: the common ancestry should be preserved. Ideally 
interpretations which agree on events both before and after a split should share 
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the sane representation for them; this is called a join. 

The datachart serves these functions. Like the planchart, the datachart 
is a chart, so that it can economically store common substructure. Suppose that 
two interpretations have identical assignments for the first H events, and then 
split. The split corresponds to a single node having two descendants. 
Assertions corresponding to the shared part of the interpretation are 
automatically inherited from the parent node (figure 11:8). 

Whenever a low plausibility event assignment occurs the following actions 
are performed: 

lm ^/innlVt "/ 5 K dd l d at l he ° Urrent node ' indica *ing which event 

So«?mH« f?, 0Ut *l be made - This ensures that ^e «me 
possibilities will not be repeatedly pursued. 

2 * thl W n . n r°p d nt *«* h^ ^' WMch wil1 inherlt P rior assignments from 

,mr*r a rr °^ ' ThiS ensures that cnan 9 es » hi <* reflect the 

^ tnTp^t ZT™* ^ " 0t affeCt the Stat ° ^formation of 

3 * InL^ Certain assi 9 nment is Performed at the new node. The 

?Sr^H°H Pe K rat \° nS associat «d with event interpretation 
(described below) are carried out. 

4 ' ?hl n I! "° de I!! Pl f ed ° n a list of NEW P artial interpretations. 

s^T^siVir 111 be scheduled for at ieast ° ne cycie 0f 

5 * SLiSTIL* r° de }\ re - exani "^ to determine if additional nodes 
\fll th. h Ut6d representin 9 alternative event assignments. 
If so, the above sequence of operations is carried out for each 
When no further alternatives seem worth considering at the 

L"Tetat 1 i m o e ns. the ^^ "° di iS PlaC6d °" a list of HUNG 

This technique has the feature that it is not necessary to explicitly 

list all of the possible alternative interpretations for a given event. After 

sprouting, the parent node no longer represents a single partial interpretation, 

but an indefinite number of implicit alternatives to its current offspring. Even 

^ after it is HUNG, the parent node contains the necessary state information to 
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generate additional possibilities if these are ever needed. 

7.4. Incrementa l Planchart Expansion 

Consider the situation in which an active partial interpretation cannot 
find an acceptable planchart assignment for its next event. Two conclusions are 
possible: either (a) the current partial interpretation is a dead end, and 
should be moved to the HUNG list; or (b) the current partial interpretation is 
viable, but the planchart has not been expanded sufficiently to account for the 
current data. 

This decision is crucial. If PAZATO is too miserly in allowing planchart 

growth, an event could be mis-interpreted as a deviant version of an existing 

leaf, when only slight growth would have allowed it to match a new leaf exactly. 

But if PAZATN is too eager to expand the planchart, the number of irrelevant 

/f m s solutions proposed could be enormous. 

This decision is also very difficult, being complicated by the 
circumstance that data events need not identically match planchart leaves: they 
can differ because of postulated bugs or variant but acceptable parameter values 
(such as scale factors). 

Four techniques are germane to this decision and its complications. 

1. Protocol events are converted to a canonical form. This allows 
for handling simple forms of equivalence such as FORWARD -100 
versus BACK 100. 

2. Standard spelling correction procedures 7 are applied to 
unrecognized protocol events, using the fringe of the planchart 

mis?ellings nary ' ™ S ^^ f ° r handlin9 simple "^typings and 

3. A hash coding scheme uses the critical terms of an event (e a 

the FORWARD but not the 100) as keys. 8 This allows acceptable 

J" 5 / events (e.g.,, those differing only by a scale 
factor) to be located. 

r\ 4. The neighbors of a planchart leaf provide expectations which 
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7.5. Markers and M arker Prop agatinn 

A marker propagation technique helps in deciding whether to expand the 
Planchart by providing a precise representation for expectations. Markers also 
determine the final protocol parse by selecting a pathway through the planchart. 
Assigning a protocol event to a planchart leaf marks that leaf. Three types of 
markers are used: (1) . standard narker for fVMti that ^ ld-ntlcally or 
differ only in a flexible parameter value; (2) a distinguished marker for top 
down DEFINED plans prior to encountering the body of the subprocedure; and (3, a 
distinguished marker for deviant events involving mistypings or similar errors. 

A constituent is expected to the extent to which finding it results in 
propagations, where propagation through the planchart is characterized by rules 



such as 

MPR-DISJ 



O 



™nA f It* Parent 0f a aarked node ls disjunctive (i e a 
split), the parent is marked; ■ t 1 -®-. a 

MPR-CONJ. If the parent of a marked node is conjunctive (e a sfoi an n 
every sibling of the marked node is market tU 'parent Ts 

The rules shown here are incomplete. Top down DEFINED plans, for 
example, receive special treatment to ensure that after completing a 
superprocedure the expectations for its subprocedures remain in effect. 

As an example of the use of these rules, consider a bottom-up DEFINED 
Plan, where a subprocedure is first defined and then called by a superprocedure. 
After the subprocedure definition has been encountered, its use by some 
superprocedure is expected. The planchart would contain a marked SOLVE node for 
the subprocedure and an unmarked USE node for its use in the other procedure. 
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both dominated by an unordered conjunctive «&« node (figure 11:9). The USE is 
expected because marking its node would result in a propagation at least as far 
as the SOLVE dominating the DEFINED node. 

Suppose that an expectation (such as the bottom-up DEFINED plan example) 
fails to be satisfied after many events. One possibility is that the partial 
interpretation which expects it is completely wrong, and should be abandoned. A 
second possibility is that the partial interpretation is basically correct, but 
the student has accomplished the expected effect in an alternative way (e.g., 
incorporated the subprocedure's definition in-line instead of calling it as 
expected). This second case turns off the expectation, since it becomes 
dominated by a marked node (figure 11:10). 

A third possibility is that the student accidentally left out the 
relevant line of code. This is detected when protocol events indicate that the 
episode is finished. In the Logo world this corresponds to encountering the END 
statement for the superprocedure. END statements force propagations even when 
some expectations are not satisfied; but the plan is flagged as incomplete, 
debugging expectations are generated, and the plausibility is lowered. If the 
debugging predictions are then confirmed, the plausibility is restored and the 
expectation considered satisfied. 

Markers, as a representation for expectations, provide evidence regarding 
the plausibility of interpretations, which is especially useful when planchart 
expansion is under consideration. Typical plausibility guidelines include: 

PLG-1. Event assignments that result in longer chains of prooaaation* 

ZLTnlt- PlaUSible than th0Se that result in shorter chain, S 
propagations or none at all. 5 OI 

PLG ' 2 ' P?auXe at tn°an S thofa li'lV" eXpectations ""satisfied are more 
plausible than those that leave many expectations unsatisfied. 
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(already 
marked) 
X 

"SOLVE (WELL) — •• 




. . .SOLVE (*WELL*)- 






•DEFINED- 



& 



/"N 



USE (WELL) ••• 



A use of WELL is expected because it would cause 
the propagations shown asf's. 



FIGURE 11:9 DEFINED PLANS: AN EXAMPLE OF PROPAGATION AS EXPECTATTOM 
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Protocol 



TO WW 



r\ 
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SOLVE (WW)— • . .SEQ 



V 

-SOLVE (*WELL*)- 



X 

-SOLVE (WELL) 
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-USE (WELL) 



X 

-SOLVE ( WELL )- 



in-line code for well 



\. 



END 



This use of WELL is no longer expected, 
since it is now dominated by a marked 
node. 

FIGURE II:1Q EXPECTATIONS CANCELLED, DOMINATED BY MARKED NODE 
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7.6. Preprocessing 

PAZATN includes a preprocessor which performs four functions. 

1. Low level syntactic anomalies such as typoaraohicai «rmr* 
corrected using the RUBOUT and BREAK keys^a^e filtered Tout' 
only the corrected versions of such events are examined? ' 

2 ' sc^^VuRTp'f?,' 10 !! ClUGS are n0ted - For •»»Pl«. ^th raster 
is reUnJ hLJ ES , CL ^ eberraan 197 6] global connectivity of vectors 
is readily detectable and suggests a segment boundary. 

3 * Ii m ,M 9 „„ dat n a a ? 5 0l J ected - ^is information may be of value in 

■ Xsibmtv y of 0l . n B 'J' ClaimS ' and in ««»■• instance, the 

b P e^en type-ins " lnterpretatlon *"•"«'« ^n the elapsed time 

4 * I!lL Pri, ! ary * yntactic class of ea <* event is recorded to avoid 
recomputing it under each interpretation. Classification i J 
performed by a separate module which can be ^invoked if the" 
primary class is later called into question. invo * ed if the 

7.7. The Event Classifier 

^ The event classifier, one of the few PAZATN modules which must be 

redefined for each domain, contains the syntactic knowledge necessary to 
distinguish various domain-specific event types. For the programming world, the 

event types include RUN events, EDIT events, and so on. In assigning an 

interpretation to an event, a variety of semantic and pragmatic evidence is 

ultimately considered by PAZATN, but the event classifier is restricted to 
syntactic evidence. 

The event classifier can be invoked in three modes. Normally it is 
invoked by the preprocessor, with its input an event and its output the event's 
primary syntactic class; for most events, this is sufficient. In the second 
mode it is invoked by partial interpretations which question the primary 
syntactic class, with a specific alternative class being considered. Here its 
input is an event and a class name; its output is a numerical score summarizing 
^ the syntactic evidence supporting the alternative class. In the third mode the 
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classifier is invoked by partial interpretations which question the primary 
syntactic class but with no specific alternative class under consideration. An 
exhaustive rank ordered list of categories and scores is returned. 

Event classification will be performed using straightforward pattern 
matching. No further details are given here. 

7.8. The Event Interpreter 

The event interpreter is responsible for category independent operations 
of event interpretation. This includes the node sprouting sequence described in 
the datachart section, the processing required for marker propagation, and the 
Plausibility computations. The rationale for grouping these activities is 
modularity: they are required for every category of event interpretation. 

The event interpreter is PAZATN's inner loop. It is invoked by the 
^ scheduler with two arguments: a partial interpretation, and a protocol event. 

It attempts, in cooperation with one or more event specialists, to account for 
the protocol event in the context of the partial interpretation. This can result 
in the creation of additional (descendant) partial interpretations. Control 
returns to the scheduler when event interpretation is complete. 

7.9. The Event Specialists 

A collection of domain specific event specialists (ESP's) are responsible 
for category dependent operations of event interpretation. Each specialist 
contains the requisite knowledge for analyzing events of a particular syntactic 
type. The event interpreter invokes an ESP. in the context of a partial 
interpretation, with an event and an implicit assumption regarding its syntactic 
category. The specialist is free to assign any interpretation to the event which 
is consistent with the category. 

^ If the event ^ecialist does not return with a sufficiently plausible 
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event assignment, the event interpreter then considers the possibility that the 
syntactic category postulated for the event is incorrect. Whenever an event is 
interpreted as being in error, expectations for diagnosis and repair are 
generated by DAPR at the request of the event interpreter. 

ESP's use a decision tree organization to factor the analysis into 
several cases. Each case represents an assumption about intent; if the 
assumption is uncertain, the state of the interpretation is preserved by 
sprouting a new datachart node. This is exemplified by the Logo ADDCODE ESP, 
whose flowchart appears in figure II: 11.'° 

The ADDCODE classification assumes that the current event is intended to 
add a new line of code to the procedure definition. Hence it must be determined 
whether Logo is actually in definition mode. If not, the following event will be 
an error message. If the ADDCODE assumption is correct despite the error, the 
current event will be repeated after a TO event. 

Lookahead is required to assign the current event to be an erroneous 
version of a later event. However, in a real time tutoring application, the 
later event might not have occurred yet; moreover, processing more than one 
event would exceed the scheduler's resource allocation. This dilemma is resolved 
by creating a demon to represent the current event assignment. The demon will 
fire when the future event is assigned, assigning the now current event to be a 
deviant version of the later event. 

In the case where Logo is in definition mode, ADDCODE branches to one of 
the following subcases: (a) the added code is a turtle primitive; <b) the added 
code is a Logo control statement (such as a recursion or iteration line); (c) 
the added code is a call to a user procedure other than a recursion line. 
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FIGURE II; 11 FLOWCHART FOR ADDCODE ESP 



r*\ 



fs 



Protocol Analysis 2.33 Miller & Goldstein 

7.10. The Scheduler 

The scheduler's task is to cause partial interpretations which have a 
reasonable likelihood of succeeding to make progress, and prevent those that are 
likely to fail from consuming valuable resources. Operationally this means 
driving PAZATN through a best first coroutine search of the space of partial 
interpretations . 

The search is accomplished by maintaining three lists of partial 
interpretations: NEW, ACTIVE, and HUNG. The scheduler cycles through the ACTIVE 
list, allowing each item to process one protocol event. Then the plausibility of 
each modified interpretation is recomputed, and the ACTIVE and HUNG lists are re- 
chosen. NEW interpretations, which result from the splitting of ACTIVE 
interpretations on the previous cycle, are then moved to the ACTIVE list, 
guarantying them at least one quantum of processing. The plausibility of a 
partial interpretation increases with each additional event accounted for. (This 
acts to decrease the relative plausibility of older HUNG interpretations.) 

This process continues until at least one ACTIVE interpretation has 
processed the last input event without unsatisfied expectations. If the first 
successful interpretation is not sufficiently better than every other candidate, 
some of the better alternatives are pursued until they become implausible or 
determine that the protocol may successfully be interpreted in more than one way. 
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8. Refining the Protocol Parser 

8.1. Lookahead 

8.2. Least Commitment 

8.3. Differential Diagnosis 

Our basic protocol parsing scheme is to generate expectations with PATN 
and then try to match these expectations to a protocol. This process is refined 
by several techniques which have enhanced the effectiveness of problem solving 
and language processing programs: lookahead (e.g., [Aho & Ullman 1972]), least 
commitment (e.g., [Sacerdoti 1975]) and differential diagnosis (e.g., 
[Rubin 1975]). 

8.1. Lookahead 

Lookahead and least commitment are related search strategies designed to 
^ avoid premature decisions based on inadequate evidence, which can result in 

needless backup. Lookahead consists of briefly examining subsequent input events 
before interpreting the current event. 

PAZATN can accomplish a limited form of lookahead by using demon 
procedures to represent event assignments. When the current event assignment 
depends upon a future event assignment, a demon is created which will complete 
the current assignment when the missing evidence from the future assignment is 
available. 

8.2. Least Commitment 

Variability in solution order exemplifies the need for avoiding premature 

commitments. PATN always defines the top level plan before expanding 

subproblems, representing strict top-down problem solving (figure 11:12), but 

human programming is rarely this uniform. When the need for a particular subgoal 

r^ has been established, it may be expanded immediately, prior to completing the top 
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SOLVE (WW) 



SEQ 

SOLVE (*ROOF*) 
DEFINED 
USE 
SOLVE (ROOF) 



SOLVE (*POLE*) 

DEFINED 

USE 

SOLVE (POLE) 



SOLVE (*WELL*) 
DEFINED 

USE 




?TO WW 
>10 ROOF 
>20 POLE 
>3 WELL 
>END 

?TO ROOF 



>END 
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SOLVE (WELL) 



I >END 



?T0 WELL 



>END 



FIGURE II; 12 A TOP-DOWN EXPANSION FOR WISHINGWELL 
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level plan, representing bottom-up problem solving (figure 11:13). 

Least commitment helps to minimize misleading mismatches between 
Planchart and protocol resulting from different solution orders. This is 
accomplished by using unordered conjunctive ■&• nodes in the planchart. Thus 
when DEFINED plans are expanded to USE & SOLVE, the SOLVE may occur prior to the 
USE with no loss in plausibility. 

The least commitment policy is applied to variability in invocation order 
as well. When, as was the case with WW, more than one invocation order is 
acceptable, the planchart is split. This parallels the use of procedural nets 
[Sacerdoti 1975] to avoid overspecifying ordering constraints (figure 11:14). 
The chart data structure allows the ambiguity to be represented without 
significant additional cost: if the mainsteps are identical for both orders, 
then two copies will not be stored. 

Despite its virtues, though, least commitment could be overdone, 
resulting in so large a disjunction of expectations that no guidance would be 
obtained. PAZATN strikes a balance between overcommitting itself and refusing to 
take decisive action: it avoids arbitrary choices in the course of a given 
decomposition strategy, but adheres to a given formulation of the model unless 
required to change it by specific bottom-up evidence. 

8.3. Differential Diagnosis 

The use of demon procedures to implement lookahead was discussed earlier. 
Another use of demons is to perform differential diagnosis, using highly specific 
clues to distinguish between similar competing interpretations. The primary 
application of differential diagnosis demons is to the choice between assigning 
an event to one of an existing disjunction of expectations, and reformulating the 
problem description in response to bottom-up evidence. 
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FIGURE II; 14 A PROCEDURAL NET FOR BUILDING A TOWER 
AFTER CRITICISM TO RESOLVE CONFLICTS 

[BASED ON SACERDOTI, 1975, p. 15] 
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To illustrate this, we present one example of a complementary pair of 
demon templates. These templates can be instantiated to realize differential 
diagnosis behavior in specific situations. 

A. If the current code segment (or its picture) matches a 
disjunctive subset of the current expectations, select that 
subset. 

B. If no expectation matches the current code segment (or its 
picture), consider a reformulation using the segment's effect as 
a subgoal. 
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9. Conclusion 

9.1. Recapitulation 

9.2. Implementation Plans 

9.1. Recapitulation 

In this report we have investigated the problem of analyzing elementary 
problem solving protocols. The result of this investigation is the design for 
PAZATN, a domain independent protocol parsing scheme, which was applied to the 
Logo graphics programming domain. Coupled with the Logo ESP's, the design was 
sufficiently well-specified that PAZATN could be hand-simulated for a simple 
example with encouraging results. The foundation for the approach was SPADE, a 
linguistic theory of design in which problem solving is viewed as a structured 
process of planning and debugging. This led us to the definition of an 
interpretation as a parse tree augmented by semantic and pragmatic annotation 
associated with each node. 

A key ingredient in the design is a machine problem solver called PATN. 
PATN employs an augmented transition network to represent fundamental planning 
concepts, including techniques of identification, decomposition, and 
reformulation. Considerable leverage is obtained from PATN's ability to generate 
successively less preferable solution paths, by a series of pragmatically guided 
Planning decisions, as well as from PATN's characterization of certain bugs as 
errors in these planning choices. 

We found an analogy to computational linguistics to be fruitful, 
providing insights into data representations and search strategies which are 
characteristic of research in syntactic analysis [Kay 1973; Kaplan 1973] and 
speech recognition (e.g.. [Lesser et al. 1975; Paxton & Robinson 1975]). For 
example, the ckart representation is used to economically store well-formed 
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substructures. Lookahead, least commitment, and differential diagnosis are 
example strategies used to refine PAZATO's search for a parse, These allow for 
proceeding on the basis of reasonable assumptions when necessary, while retaining 
the ability to modify the interpretation in response to anomalies. 

The analysis procedure has been designed to obtain maximal advantage from 
both top-down guidance from the task description and bottom-up protocol evidence. 
Analysis proceeds by a best first coroutine search of a space of partial 
interpretations. The planchart, a data structure resembling an AND/OR goal tree. 
is used to keep track of expectations. By careful selection of the 
representational scheme, this structure achieves considerable storage economy. 
Partial knowledge of structure and of the status of expectations is recorded 
using a scheme of planchart markings and marker propagations. The planchart is 
incrementally expanded by PATN when existing expectations are inadequate in view 
of the protocol data. A second chart, the datachart, is used to keep track of 
the state of alternative partial interpretations. 

Although PAZATN is not yet a working program, the design is sufficiently 
specific so as to be hand-simulable. In hand-simulation, there is a danger of 
unintentionally drawing upon knowledge which has not been isolated or formalized. 
Care was exercised to avoid this pitfall, and the examples are encouraging 
evidence that the approach is fundamentally sound. Still, hand-simulation is not 
seen as a substitute for implementation. The next phase of the research is to 
implement and experiment with a prototype analyzer. 

PAZATN is a generalization and extension of previous approaches. PAZATN 
grew out of Goldstein's [1974; 1975] plan-finder for MYCROFT. The differences 
are that PAZATN: (a) generates interpretations consistent with the recently 
developed SPADE theory; (b) handles the wider range of event types necessary to 
analyze protocols rather than finished programs; (c) takes advantage of the 
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dynamic information in these additional event types regarding subgoal structure 
and development; and «t) is not limited to the Logo domain. The SPADE theory 
developed from the MYCROFT theory of program understanding as well as related 
work by Sussman [1973], Papert [1971] and Sacerdoti [1975]. [Goldstein & 
Miller 1976b] argues that SPADE represents progress over this earlier theorizing. 
PAZATN also complements the features and limitations of analyzers developed at 
Carnegie-Mellon University. The major theoretical advance is a highly structured 
model of program synthesis. The major practical advance is the modularization of 
domain specific knowledge, which indicates that the PAZATN framework ought to be 
applicable to a wide variety of task domains. 

PAZATN is independent of the detailed form of the synthetic formalism: 
it does not intrinsically depend on PATN being an augmented transition network. 
It is only necessary that the synthetic component plan and debug by making a 
series of pragmatic choices which can be summarized by the planchart data 
structure, and that it be capable of generating not one, but an entire space of 
progressively less favored solution paths. Finally, an implicit assumption runs 
throughout the analyzer's design that solutions can be decomposed into syntactic, 
semantic, and pragmatic elements. It may be that any synthetic formalism 
satisfying these constraints is trivially equivalent to an ATN. Such questions 
are notoriously difficult to settle. 

However, an important issue in the design is the breadth of the synthetic 
theory. There are of course particular omissions such as conditional plans, 
which have been deliberately ~ but only temporarily ~ ignored. The greater 
threat comes from the unknown. Even very young children display incredible 
richness in their problem solving. Although SPADE's origins are partly 
empirical, crucial phenomena - perhaps those most in need of investigation - 
may have been overlooked. This remains a topic for investigation. 
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9,2. Implementation Plans 

There are several ways in which an apparently sound design could fail in 
implementation. The space of partial interpretations could turn out to be very 
large relative to the sources of constraint which have been isolated. The 
variety of knowledge used by human programmers could greatly surpass our current 
estimates. PAZATN's storage requirements could exceed practical bounds. The 
analyzer could be too rash in its heuristic quest for efficiency, terminating 
prematurely with unacceptable interpretations. Too great a demand could be 
placed on PATN's ability to find reformulations encompassing bottom up evidence. 
Hand-simulations or even partial implementations could overlook such impediments. 

Consequently, complete implementation of a prototype system is essential 

for validating the research. We intend to perform this implementation 

incrementally, beginning with an interactive version. At first, only 

straightforward bookkeeping functions will be automated. The computer will 

record plans and event assignments using the two charts, but decisions regarding 

which interpretations to pursue will be made by a human investigator. This will 

be replaced by a version which performs the routine analysis of most events, only 

requesting help on more difficult cases. Eventually, the analysis will be 

handled completely by the machine. A modeling component for inducing 

personalized ATN's will be implemented, and its predictive power explored. To 

demonstrate PAZATN's understanding of the protocols, a question answering module 

using a formal query language will be constructed to operate over PAZATN's 

output. 
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10. Notes to Book II 

~. .. „ \' CBrown et - al - 19 ?7] includes a chapter indicating the direction of 
our efforts to apply PAZATN to the symbolic integration domain Erection of 

nistvninfl 2 :, ™* one e "eption to this is that some irrational errors (such as 
mistypings) can be recognized as unsuccessful manifestations of PATN plans. 

unH«r«i- a „ 3 H<„I hiS n USe ° f dem ° ns is lns P ired b y Charniak's [1972] work on story 
understanding. Demons are a type of antecedent theorem [Hewitt 1972]. 

4. The chart data structure is due to Kay [1973] and KaDlan riQ7*i 

ou^znVLTT 9r T S il rK & Sir ° Vich 1976] are ^ta structures s^miar Jo 
Jhink that - Lil«\l** r °\? h * b t Si \° f independe "t fo ™l considerations. We 
think that, because of the extension from trees to charts as well as the 
incorporation of a larger variety of node types, our plancharts are an 

iSrJafSnn ^ ge , neraliZed ^^ ^aph, along the dimension o generality? 
storage economy, and expressive power. «««*• a.ni.jr, 

mn mn„^ 5 ^ W8 ^ 1 ^ tend t0 P rovide p AZATN with limited heuristics for recognizing 

r if^f 5, H T Ver ' relying on user chosen ™**s ^r guidance in 
general would be too unreliable. Hence, to emphasize that such guidance can be 
dispensed with, we assume here that procedure names are unrecognizable 

*„a *w 6 \ In a f si 9 nin 9 a Protocol event to a planchart leaf, the type of event 
fl e H v r alue of : MODEL are considered, but the other semantic varices an \T* 
pragmatic assertions are generally not considered. This is a simplification 
which ignores the possibility for complex semantic and pragmatic ambiguities 

ADVIC^^^^^ be identiCal eXCept for the valuaJf 

.ADVICE at some node. Although this difficulty seems unlikely PAZATN could ha 

elaborated slightly to handle it. Here we ignore the problem and show only the 
structure description and the name of the submodel in our diagrams? ^ other 
stressed.) 3 pra S Qatic assertions, being assumed unambiguous, are 

7. Interlisp [Teitelman 1974, pp. 17.10-17.14] orovides such a «:n«»inn„ 
corrector. See also [Teitelman 1970]. Provides such a spelling 

al. 1967, 8 pp SUC 806-8S h 7]! qUeS '" *" ^^ "^ ***' f0r eXample ' W«»°l«tt et 

tvn„ -in „ 9 r' !° r example \ if mucn mor e than the typical time elapses between the 
type-in of two consecutive events, it is more plausible to interpret the second 
event as initiating a new episode. A more specific example involves the freaSenJ 
errors associated with Logo line numbers. There are two «urt errors- ?S 
failing to include a line number when it is needed; and (2) accidentally 
including a line number when it is not needed. Consider the second If mJch 

tvoe 2 a of t ?L tyPi " 1 H tl " e , el i: PSeS b8tWeen the type " in of the line nui'er and the 
type-in of the remainder of the event, it becomes more plausible to interoret tha 

RatnIr a th a „ bU ? 9yRUN 6Vent "^ tha " a l ^ 1 ^ 1^11^1.^1^ 
Rather than storing every value, however, the preprocessor will aerumni 2*1 
/-N summary statistics, only recording the sp'ecific'dafa f^r typ^-in's whS "III 
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markedly slower than the average. 
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